Skip to contentSkip to conversation

Continuous Integration Build Failures

DISCUSSION Johnathan Mantey 5 participants 8 messages Oct 24, 2019 – Oct 25, 2019

lore
Share & tools
lore
#0 lore raw

I would like to propose a change to how the continuous integration system works.

I understand there are many builds, and there is a lot of data associated with the builds. Thus the current desire to remove the log file data in a short amount of time is a requirement. This works alright when the build succeeds. It's unhelpful when the build fails. Identifying where the build fails is impossible after approximately an hour. As an ordinary contributor I don't know how to make the CI system rebuild the source code so that the log files are available again without pushing some new change that consists of a useless piece of whitespace (or some other pointless change).  It shouldn't be necessary for the contributor to make requests for a build restart to the CI maintainers, they have their own agenda.

Are the maintainers of the CI system willing to make a change that aids in debug?

Suggestions:

1. Don't delete the log on build fails.
2. Delete everything but the log ascii output on build fails.
3. Email the ascii logfile for build fails
4. Email a compressed debug bundle to the submitter?
5. Allow build fails to be restarted by the submitter so the logs can be regenerated, inspected, and captured.
6. other...?

Signature · 6 lines
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: [email protected] <mailto:[email protected]>
#2 lore raw
In reply to Johnathan Mantey
Quoted 7 lines · Johnathan Mantey

I would like to propose a change to how the continuous integration system works.

I understand there are many builds, and there is a lot of data associated with the builds. Thus the current desire to remove the log file data in a short amount of time is a requirement. This works alright when the build succeeds. It's unhelpful when the build fails. Identifying where the build fails is impossible after approximately an hour. As an ordinary contributor I don't know how to make the CI system rebuild the source code so that the log files are available again without pushing some new change that consists of a useless piece of whitespace (or some other pointless change). It shouldn't be necessary for the contributor to make requests for a build restart to the CI maintainers, they have their own agenda.

Are you referring to the bitbake CI jobs or the repository CI jobs?

Quoted 10 lines

Are the maintainers of the CI system willing to make a change that aids in debug?

Suggestions:
• Don't delete the log on build fails.
• Delete everything but the log ascii output on build fails.
• Email the ascii logfile for build fails
• Email a compressed debug bundle to the submitter?
• Allow build fails to be restarted by the submitter so the logs can be regenerated, inspected, and captured.
• other...?

At first glance these are all good ideas. Andrew how many of these can Jonathan implement himself and how many of them require access to the Jenkins instance?

#3 lore raw

Brad,
No I had not seen that doc.  That said, I worked with someone here to run the CI build locally.  I got the Docker instance to perform a build on the submission that instigated this email trail. The Docker instance passed on my code changes.  Yet, the upstream Gerrit build does not. Now I don't have logfiles on the upstream Gerrit server to find out why.  I believe I've done my due diligence for preparing the code for a successful build.  At the end of the day the only place that matters for build success/failure is the upstream system.  As such anything that can be done to improve that system is, in my opinion, a benefit to the community.

My specific issue is:
https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-networkd/+/24666
I have no idea why this is failing now that the logs have been deleted, and I can't determine a way to manually start a new build to get logs without pushing meaningless commits.  I'd rather not waste reviewers time having to see email messages about meaningless commits.

Quoted 18 lines · Brad Bishop
Quoted 6 lines · Johnathan Mantey

I would like to propose a change to how the continuous integration system works.

I understand there are many builds, and there is a lot of data associated with the builds. Thus the current desire to remove the log file data in a short amount of time is a requirement. This works alright when the build succeeds. It's unhelpful when the build fails. Identifying where the build fails is impossible after approximately an hour. As an ordinary contributor I don't know how to make the CI system rebuild the source code so that the log files are available again without pushing some new change that consists of a useless piece of whitespace (or some other pointless change). It shouldn't be necessary for the contributor to make requests for a build restart to the CI maintainers, they have their own agenda.

Are you referring to the bitbake CI jobs or the repository CI jobs?

Quoted 9 lines

Are the maintainers of the CI system willing to make a change that aids in debug?

Suggestions:
• Don't delete the log on build fails.
• Delete everything but the log ascii output on build fails.
• Email the ascii logfile for build fails
• Email a compressed debug bundle to the submitter?
• Allow build fails to be restarted by the submitter so the logs can be regenerated, inspected, and captured.
• other...?

At first glance these are all good ideas. Andrew how many of these can Jonathan implement himself and how many of them require access to the Jenkins instance?

Signature · 6 lines
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: [email protected] <mailto:[email protected]>
#4 lore raw
Quoted 6 lines · Johnathan Mantey

Brad,
No I had not seen that doc. That said, I worked with someone here to run the CI build locally. I got the Docker instance to perform a build on the submission that instigated this email trail. The Docker instance passed on my code changes. Yet, the upstream Gerrit build does not.

You and any other contributor should be able to reproduce the CI build locally. If that isn’t the case, that is a bug somewhere. I’ll try downloading your patch.

Quoted 1 line

Now I don't have logfiles on the upstream Gerrit server to find out why. I believe I've done my due diligence for preparing the code for a successful build. At the end of the day the only place that matters for build success/failure is the upstream system. As such anything that can be done to improve that system is, in my opinion, a benefit to the community.

I agree.

Quoted 2 lines

My specific issue is: https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-networkd/+/24666
I have no idea why this is failing now that the logs have been deleted, and I can't determine a way to manually start a new build to get logs without pushing meaningless commits. I'd rather not waste reviewers time having to see email messages about meaningless commits.

Yep, that makes sense and I can understand that point of view. We all want to work independently. I’ll just say that while we work to make the process better, I have no problem with you or anyone else pinging me in IRC to go kick a job where the logs have wrapped.

-brad

#5 lore raw
In reply to Johnathan Mantey
Quoted 5 lines · Johnathan Mantey

Brad,
No I had not seen that doc.  That said, I worked with someone here to run the CI build locally.  I got the Docker instance to perform a build on the submission that instigated this email trail. The Docker instance passed on my code changes.  Yet, the upstream Gerrit build does not. Now I don't have logfiles on the upstream Gerrit server to find out why.  I believe I've done my due diligence for preparing the code for a successful build.  At the end of the day the only place that matters for build success/failure is the upstream system.  As such anything that can be done to improve that system is, in my opinion, a benefit to the community.

My specific issue is: https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-networkd/+/24666
I have no idea why this is failing now that the logs have been deleted,

You can login to the Jenkins instance using your github credentials and re-trigger the build.

Quoted 29 lines

and I can't determine a way to manually start a new build to get logs without pushing meaningless commits.  I'd rather not waste reviewers time having to see email messages about meaningless commits.

Quoted 17 lines · Brad Bishop
Quoted 5 lines · Johnathan Mantey

I would like to propose a change to how the continuous integration system works.

I understand there are many builds, and there is a lot of data associated with the builds. Thus the current desire to remove the log file data in a short amount of time is a requirement. This works alright when the build succeeds. It's unhelpful when the build fails. Identifying where the build fails is impossible after approximately an hour. As an ordinary contributor I don't know how to make the CI system rebuild the source code so that the log files are available again without pushing some new change that consists of a useless piece of whitespace (or some other pointless change). It shouldn't be necessary for the contributor to make requests for a build restart to the CI maintainers, they have their own agenda.

Are you referring to the bitbake CI jobs or the repository CI jobs?

Quoted 9 lines

Are the maintainers of the CI system willing to make a change that aids in debug?

Suggestions:
• Don't delete the log on build fails.
• Delete everything but the log ascii output on build fails.
• Email the ascii logfile for build fails
• Email a compressed debug bundle to the submitter?
• Allow build fails to be restarted by the submitter so the logs can be regenerated, inspected, and captured.
• other...?

At first glance these are all good ideas. Andrew how many of these can Jonathan implement himself and how many of them require access to the Jenkins instance?

Signature · 6 lines
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: [email protected] <mailto:[email protected]>
#6 lore raw
In reply to Brad Bishop
Quoted 19 lines · Brad Bishop
Quoted 8 lines · Johnathan Mantey

I would like to propose a change to how the continuous integration system works.

I understand there are many builds, and there is a lot of data associated with the builds. Thus the current desire to remove the log file data in a short amount of time is a requirement. This works alright when the build succeeds. It's unhelpful when the build fails. Identifying where the build fails is impossible after approximately an hour. As an ordinary contributor I don't know how to make the CI system rebuild the source code so that the log files are available again without pushing some new change that consists of a useless piece of whitespace (or some other pointless change). It shouldn't be necessary for the contributor to make requests for a build restart to the CI maintainers, they have their own agenda.

Are you referring to the bitbake CI jobs or the repository CI jobs?

Quoted 5 lines

Are the maintainers of the CI system willing to make a change that aids in debug?

Suggestions:
• Don't delete the log on build fails.

Other then some old unsupported jenkins plugins, I’m not seeing any obvious way to this one.

Quoted 1 line
Quoted 1 line

• Delete everything but the log ascii output on build fails.

The CI jobs really don’t have much, there are no artifacts associated with them. Lets try delaying the build deletes for 5 days and see if anything horrible happens. I just changed this in the job.

Quoted 1 line
Quoted 1 line

• Email the ascii logfile for build fails

I don’t believe we have email support on our Jenkins instance currently but if we did, this could be possible. The email address is known via the gerrit plugin env variables. There seem to be some jenkins plugins for sending emails with data on failures.

Quoted 1 line
Quoted 1 line

• Email a compressed debug bundle to the submitter?

See above

Quoted 1 line
Quoted 1 line

• Allow build fails to be restarted by the submitter so the logs can be regenerated, inspected, and captured.

Sounds like if you are logged into jenkins you can resubmit the job, but that’s probably not all that useful because you could just look at the fail. What a user really wants is to be able to retrigger the CI build from within gerrit. Maintainers can just add a +1 to Ok-To-Test to cause this but non-maintainers can not. The current “Ok-To-Test” design is in place in gerrit to ensure only trusted people are running code on our CI infrastructure. We can’t just let anyone with a github id approve the CI runs. If there was a way to tie that ability to the user being in one of the gerrit CI groups then that would be great. I don’t have much experience in that area though.

Quoted 3 lines
Quoted 1 line

• other…?

At first glance these are all good ideas. Andrew how many of these can Jonathan implement himself and how many of them require access to the Jenkins instance?

Unfortunately I think all require access to the jenkins instance and for some access to the actual VM (email support).

#7 lore raw
Quoted 16 lines · Andrew Geissler
Quoted 16 lines · Brad Bishop
Quoted 8 lines · Johnathan Mantey

I would like to propose a change to how the continuous integration system works.

I understand there are many builds, and there is a lot of data associated with the builds. Thus the current desire to remove the log file data in a short amount of time is a requirement. This works alright when the build succeeds. It's unhelpful when the build fails. Identifying where the build fails is impossible after approximately an hour. As an ordinary contributor I don't know how to make the CI system rebuild the source code so that the log files are available again without pushing some new change that consists of a useless piece of whitespace (or some other pointless change). It shouldn't be necessary for the contributor to make requests for a build restart to the CI maintainers, they have their own agenda.

Are you referring to the bitbake CI jobs or the repository CI jobs?

Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

Are the maintainers of the CI system willing to make a change that aids in debug?

Suggestions:
• Don't delete the log on build fails.

Other then some old unsupported jenkins plugins, I’m not seeing any obvious way to this one.

It's a valid point, I have seen people submitting changes to gerrit and got a build failure, but could not find the detailed log why it fails.
Currently, there is only console logs on Jenkins, but the build/test logs are not accessible.
Is there a way to post the build logs and test logs to a place, e.g. Dropbox/GoogleDrive/etc, so that it could be persistent for a longer time?